Interaction between a viewer and an object in an augmented reality environment

ABSTRACT

A method includes: triggering rendering of an augmented reality (AR) environment having a viewer configured for generating views of the AR environment; triggering rendering, in the AR environment, of an object with an outside surface visualized using a mesh having a direction oriented away from the object; performing a first determination that the viewer is inside the object as a result of relative movement between the viewer and the object; and in response to the first determination, increasing a transparency of the outside surface, reversing the direction of at least part of the mesh, and triggering rendering of an inside surface of the object using the part of the mesh having the reversed direction, wherein the inside surface is illuminated by light from outside the object due to the increased transparency.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of, and claims priority to, U.S.application Ser. No. 15/820,813, filed on Nov. 22, 2017, and entitled“INTERACTION BETWEEN A VIEWER AND AN OBJECT IN AN AUGMENTED REALITYENVIRONMENT”, the disclosure of which is incorporated by referenceherein in its entirety.

TECHNICAL FIELD

This document relates, generally, to interaction between a viewer and anobject in an augmented reality environment.

BACKGROUND

Some systems that provide augmented reality (AR) environments do so bygenerating a view that includes both an image of a physical environment(e.g., captured using a video camera function on a device) and one ormore objects or other AR features that are added to the image of thephysical environment. Observing such an AR environment can give the userthe realistic impression of being in and/or traveling through a worldthat is a mixture of physical and AR objects. The interaction betweenthe user and these objects can be an important aspect of the user'sexperience of the AR environment.

SUMMARY

In a first aspect, a method includes: triggering rendering of anaugmented reality (AR) environment having a viewer configured forgenerating views of the AR environment; triggering rendering, in the ARenvironment, of an object with an outside surface visualized using amesh having a direction oriented away from the object; performing afirst determination that the viewer is inside the object as a result ofrelative movement between the viewer and the object; and in response tothe first determination, increasing a transparency of the outsidesurface, reversing the direction of at least part of the mesh, andtriggering rendering of an inside surface of the object using the partof the mesh having the reversed direction, wherein the inside surface isilluminated by light from outside the object due to the increasedtransparency.

Implementations can include any or all of the following features. Thedirection can be defined by a normal vector of the mesh, and reversingthe direction of at least part of the mesh can include inverting thenormal vector for the part of the mesh. The first determination caninclude determining that the viewer clips the outside surface of theobject. A material can be defined for the object, the material can beapplied to the mesh, and the method can further include, in response tothe first determination, applying a shader and texture to the material.The shader can be a Fresnel shader. The shader can be at least one of aglass shader or a water shader. A material can be defined for theobject, the material can be applied to the mesh, and the method canfurther include, in response to the first determination, applying atranslucency to the material. A material can be defined for the object,the material can be applied to the mesh, and the method can furtherinclude, in response to the first determination, altering a property ofthe material. The property can include at least one of a reflectiveness,a refraction, or a diffraction of the material. The method can furtherinclude: performing a second determination, after rendering the insidesurface, that the viewer is outside the object as a result of themovement; and in response to the second determination, restoring thetransparency of the outside surface, and reverting the direction of thepart of the mesh so the mesh is rendered outside of the object.

In a second aspect, a non-transitory storage medium can have storedthereon instructions that when executed are configured to cause aprocessor to perform operations. The operations can include: renderingan augmented reality (AR) environment having a viewer configured formovement to locations in the AR environment and for generating views ofthe AR environment from the locations; rendering, in the AR environment,an object with an outside surface visualized using a mesh having adirection defined, wherein the direction is away from the object;performing a first determination that the viewer is inside the object asa result of the movement; and in response to the first determination,increasing a transparency of the outside surface, reversing thedirection of at least part of the mesh, and rendering an inside surfaceof the object using the part of the mesh having the reversed direction,wherein the inside surface is illuminated by the light source due to theincreased transparency.

In a third aspect, a method includes: triggering rendering of anaugmented reality (AR) environment having a viewer configured forgenerating views of the AR environment; triggering rendering of anobject at a position in the AR environment; performing a firstdetermination that the viewer and the object contact each other as aresult of relative movement between the viewer and the object; inresponse to the first determination, repositioning the object in the ARenvironment based on continued relative movement between the viewer andthe object; performing a second determination that the repositioning ofthe object based on the continued relative movement reaches a threshold;and in response to the second determination, relocating the objectwithout affecting the continued relative movement.

Implementations can include any or all of the following features. Thethreshold can be defined based on a length of the object in a directionof a continued movement of the viewer. The repositioning can includeperforming a linear translation of the object according to the continuedmovement. The method can further include positioning, in response to thefirst determination, a fulcrum in the AR environment, and defining apivot arm from the object to the fulcrum, wherein the repositioning ofthe object comprises rotating the object and the pivot arm about thefulcrum. The fulcrum can be defined above the object in the ARenvironment. The fulcrum can be defined below the object in the ARenvironment. The positioning of the fulcrum and definition of the armcan be performed according to a generalized behavior defined for allproperties in the AR environment. The method can further includeapplying an effect to the object in response to the first determination,and removing the effect in response to the second determination.Applying the effect can include altering a transparency of the object,and removing the effect can include restoring the transparency of theobject.

In a fourth aspect, a non-transitory storage medium has stored thereoninstructions that when executed are configured to cause a processor toperform operations. The operations include: triggering rendering of anaugmented reality (AR) environment having a viewer configured forgenerating views of the AR environment; triggering rendering of anobject at a position in the AR environment; performing a firstdetermination that the viewer and the object contact each other as aresult of relative movement between the viewer and the object; inresponse to the first determination, repositioning the object in the ARenvironment based on continued relative movement between the viewer andthe object; performing a second determination that the repositioning ofthe object based on the continued relative movement reaches a threshold;and in response to the second determination, relocating the objectwithout affecting the continued relative movement.

BRIEF DESCRIPTION OF DRAWINGS

FIGS. 1A-B show an example of rendering an inside of an AR object.

FIGS. 2A-B show an example of rendering an inside of an AR object.

FIG. 3 shows an example of reversing a direction of a mesh for an ARobject.

FIGS. 4A-C show an example of repositioning an AR object based on aviewer position.

FIGS. 5A-C show an example of repositioning an AR object based on aviewer position.

FIGS. 6-10 show examples of methods.

FIG. 11 shows an example of a computer device and a mobile computerdevice consistent with disclosed embodiments.

FIGS. 12A-C show another example of repositioning an AR object based ona viewer position.

FIGS. 13A-C show another example of repositioning an AR object based ona viewer position.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

This document describes examples of interactions between a viewer andone or more augmented reality (AR) objects in an AR environment. Suchinteractions can occur in response to the viewer moving into contactwith the AR object. In some implementations, the interaction can involveallowing the viewer to enter inside the AR object so as to render a viewof the inside of the object. For example, an effect can be applied togive the impression that the viewer is inside an object made of amaterial such as glass or water. In some implementations, theinteraction between the viewer and the AR object can involve the viewertemporarily displacing the AR object from its current position to athreshold displacement, and thereafter relocate the AR object. Forexample, the displacement can include a linear translation of the ARobject, or to treat the AR object as if suspended from a fulcrum so asto rotate the AR object about the fulcrum. Some implementations can alsoor instead be applied in a virtual-reality system, for example toprovide an immersive virtual-reality experience.

Some existing systems allow the user to enter “inside” the AR object,but the inside does not have a defined texture or appearance. As such,the application that generates the AR environment may essentially stoprendering while the user is inside the AR object. The view may becomeentirely black to the user, which may provide a lesser experience.

Some systems that have provided virtual reality-type experiences haveembellished the computer-defined object with further definitions of theobject's interior. For example, beneath the outer surface of the objectthere can be defined an additional layer that becomes visible when thecamera “clips” the object's outer surface. However, this approach mayrequire significant additional work in defining the interior features,and it may not provide the user a plausible experience of being insidethe object.

Implementations of the present disclosure can provide advancements andimprovements in the field of computer-related technology. One drawbackof existing AR systems may be that they provide interactions between theuser and AR objects that are not realistic or not plausible. Forexample, implementations can give the user who is moving inside an ARenvironment a realistic impression of venturing inside an AR object andthereafter moving out of the object again. In some implementations, thiscan be done without embellishing the model of the AR object with anymore layers or other definitions of visual content, and/or without theneed to create any extra light sources to ensure illumination of theinside of the AR object. Transparency and/or other effects can beapplied which helps reduce or eliminate any cognitive disconnect betweenthe moment when the user is viewing the AR object from the outside andwhen the user is inside the AR object. Some implementations can allowthe user to see an ambient or surrounding environment while being insidethe AR object. When the user who travels inside the AR object does notlose sight of the outer environment of the AR object in the ARenvironment, this can serve to make the experience feel more logical andbelievable for the user.

FIGS. 1A-B show an example of rendering an inside (e.g., interior) of anAR object 100. The AR object 100 can be, or be part of, any other ARobject described herein, such as in FIG. 2A-B, 3, 4A-C or 5A-C. In FIG.1A, the AR object 100 is viewed inside a frame 102 which signifies thatthe user is observing an AR environment that is generated by a computer,for example any of the devices described below with reference to FIG.11. The AR object 100 is here a statue of a person or deity, only partof which is currently visible. Part of the head and shoulders of the ARobject 100 are here viewed from an angle. The view is generated by wayof a viewer in the AR environment, which defines the viewpoint fromwhich the resulting image is generated. Here, the AR object 100 isvisualized using an outside surface 104 which when rendered according tothe position of the viewer gives the AR object 100 is currentappearance. For example, the outside surface 104 has one or moretextures defined which define the outcome of the rendering.

Assume now that the viewer is being moved relative to the AR object 100.For example, the AR environment is defined with regard to the room wherethe user is, and the AR object is generated to be visible at aparticular position in relation to that room. The user can manipulate aninput function—such as by physically moving a handheld device which iscapturing the physical environment or by activating an input control—tochange the user's position in the AR environment. For example, the useris here manipulating the AR environment to move towards the AR object100 and ultimately to move inside the AR object 100.

FIG. 1B shows an example of the AR environment when the user has movedthe viewer to be positioned inside the AR object 100. The AR object 100is viewed inside the frame 102 and is now viewed such that the user seesdirectly onto the face of the AR object 100 from its inside. Thefeatures of the AR object 100 that are visible (e.g., eyes, nose, mouth)are generated by way of an inside surface 104′ of the AR object 100. Forexample, the inside surface 104′ is visualized by rendering of a texturethat has been determined using the texture of the outside surface 104(FIG. 1A).

A transparency of the outside surface 104 (FIG. 1A) can be increased aspart of the interaction. For example, the outside surface 104 mayinitially be non-transparent, and in response to the viewer crossing(e.g., clipping) the boundary between the outside and inside surfaces ofthe AR object 200, the transparency can be increased so that the ARobject 100 becomes at least partially transparent.

One or more filters or other effects can be added to the inside surface104′. For example, and without limitation, the inside surface 104′ canbe provided with a “glass” effect that gives the user the impression ofbeing inside a statue made of glass. For example, and withoutlimitation, the inside surface 104′ can be provided with a “glass”effect that gives the user the impression of being inside a statue madeof glass. As another example, and without limitation, the inside surface104′ can be provided with a “water” effect that gives the user theimpression of being inside a statue made of water. As such, in thesituation of FIG. 1B the user may now be seeing the ambient surroundingof the AR object 100 through a semitransparent filter that resembles theouter surface of the AR object 100. This can provide a plausibleimpression of having traveled into the inside of the AR object 100 andcan give a realistic experience of looing out at the remainder of the ARenvironment through the (semi-)transparent surface of such an object.

FIGS. 2A-B show an example of rendering an inside of an AR object 200.The AR object 200 can be, or be part of, any other AR object describedherein, such as in FIG. 1A-B, 3, 4A-C or 5A-C. The AR object 200 is herea sphere that is rendered in an AR environment. A viewer 202 is definedin the AR environment. The viewer 202 is here visually represented as acamera for explanatory purposes and may not be visible to a userobserving the AR environment—rather, the viewer 202 can be a digitaltool used to define what aspect(s) of the AR environment should bevisible to the user at any given moment. As such, the user may be ableto move the viewer 202, for example by relocating a handheld device orby manipulating an input control.

The AR object 200 has an outside surface 204 that here has defined on itrespective surface features 204A-C. For example, with reference again toFIG. 1B, each of the eyes, nose and mouth of the statue in the AR object100 can be an example of the surface features 204A-C. As such, the usermay be able to see one or more of the surface features 204A-C on theoutside surface 204 depending on where the viewer 202 is currentlypositioned. For example, the surface features 204A-B are positioned onthe near side of the sphere and are therefore drawn with solid lines,and the surface feature 204C is positioned on the far side of the sphereand is therefore drawn with a dashed line.

Each of the surface features 204A-C, and the remainder of the outsidesurface 204, can be defined using respective textures that determine theappearance(s) observable by the user. In short, the AR object 200 can bedefined using a mesh, such as by an assembly of polygons or other facesthat define the surface. A material can then be applied onto the mesh,and this material can contain a collection of properties about the ARobject 200. One property of the material can be a texture that candefine, say, a color, a pattern, a transparency, and/or one or moreother visual aspects of the AR object 200. The mesh can have a directiondefined for it, such as at any given point or other locality of the ARenvironment where the mesh is defined. The direction can vary based onthe outside surface 204. For example, the surface features 204A-C herehave respective normal vectors 206A-C. The normal vectors 206A-C arecurrently defined as pointing away from the inside of the AR object 200.The normal vectors 206A-B are drawn in solid lines corresponding to thepositions of the respective surface features 204A-B, and the normalvector 206C is drawn in a dashed line corresponding to the position ofthe surface feature 204C.

An AR environment can have virtual light defined therein, so that lightimpinging on a surface makes that surface visible to a viewer or othercamera function in the AR environment. A light source can deliver lighthaving a defined direction, such as is the case with a spotlight. Insome implementations, a state of having “no light” in the AR environmentmay be recognized, which can correspond to a medium state of lightwithout any directional source. This can be considered a non-directionalglobal light or global illumination that does not have a light source.As such, in such implementations, the available light may still have theability to illuminate a surface in the AR environment.

In this example, the AR environment has light therein. In someimplementations, this can involve light with a defined direction, and/ormedium light without a directional source. For example, light sources208A-B are here defined relative to the AR object 200. Each of the lightsources 208A-B can generate light in the AR environment. The light fromthe light sources 208A-B is virtual light in the sense that it existswithin the AR environment and can there illuminate virtual objects suchas the AR object 200 or aspects thereof. Here, light rays 210A-B aregenerated by the respective light sources 208A-B and are schematicallyillustrated.

The light generated by the light sources 208A-B can affect how featuresof the AR object 200 are visible in the AR environment. For example, theAR object 200 may only have an outside appearance defined for it by wayof the textures of the surface features 206A-C and the outside surface204, and may not currently have any inside appearance because there arecurrently no textures defined for the inside of the AR object 200. Inother words, the AR object 200 may only have an outside appearancedefined for it by way of the textures of the surface features 206A-C andthe outside surface 204, and may not have any pre-defined attributesdefined by the author of the AR object 200 to produce an interiortexture of any kind. That is, if the viewer 202 were to be positionedinside the AR object in the situation shown in FIG. 2A, the user may notsee anything because no texture has been defined inside the AR object200 and the outside surface 204 may be non-transparent. Rather,textures, transparency and/or other effects can be defined for theinside surface in a streamlined way that gives the user a cognitivelymeaningful experience when traveling between the outside and inside ofthe AR object 200.

Assume here that the viewer 202 is traveling as indicated by an arrow212. That is, the viewer 202 is moving toward the outside surface 204 ofthe AR object 200. If the movement continues, the viewer 202 will atsome point in time abut (or “contact”) the outside surface 204. Forexample, contact can be defined as occurring when the point defined asthe vantage point by the viewer 202 coincides with at least one pointthat is part of the outside surface 204. In some implementations, thisoccurrence can be described in terms of the viewer 202 clipping theoutside surface 204 of the AR object 200. Accordingly, the system thatgenerates the AR environment can determine whether and when the viewer202 is inside the AR object 200 as a result of the movement of theviewer 202. In some implementations, the clipping can occur because theAR object 200 is moving and the viewer 202 is stationary. In someimplementations, the clipping can occur because the AR object 200 andthe viewer 202 are both moving. In some implementations, the clippingcan occur because the AR object 200 is moving and the viewer 202 isstationary.

One or more operations can be performed in response to the determinationthat the viewer 202 is clipping (or has clipped) the outside surface 204of the AR object 200, for example as will now be described. FIG. 2Bschematically shows the AR object 200 when the viewer 202 is inside theAR object 200. The AR object 200 is here indicated by a half-sphere toallow illustration of the inside of the AR object 200. However, the ARobject 200 can continue to be a sphere substantially as described abovewith reference to FIG. 2A also during and after the clipping. Thediameter circumference of the AR object 200 is here illustrated by wayof a dashed line to emphasize that a portion of the sphere has beenomitted for clarity.

The AR object 200 here has an inside surface 214. The inside surface 214and the outside surface 204 can be based on a common definition.Moreover, in response to determining that the viewer 202 clips theoutside surface 204, the transparency of the outside surface 204 can beincreased. The increased transparency can allow at least some light fromthe light source 208A and/or 208B to enter the inside of the AR object200. For example, light rays 210A′ are here shown as having entered theinside of the AR object 200 from the outside.

Another example of an operation that can be performed in response to thedetermination that the viewer 202 is clipping (or has clipped) theoutside surface 204 of the AR object 200 is that the direction of one ormore meshes can be reversed or otherwise altered. For example, thenormal vector of the mesh of the surface feature 204A can be inverted toform an inverted normal vector 206A′ that instead points toward theinside of the AR object 200. As another example, the normal vector ofthe mesh of the surface feature 204C can be inverted to form an invertednormal vector 206C′ that instead points toward the inside of the ARobject 200. Because the inverted normal vectors 206A′ and 206C′ arevisible by the viewer 202 in its current position inside the AR object200, this means that at least the surface features 204A and 204C are inprinciple possible to view from the inside of the AR object 200.Moreover, because the light rays 210′ have entered the inside of the ARobject due to its increased transparency, there is now illuminationinside the AR object 200 that allows the viewer 202 to see the surfacefeatures 204A and 204C. As such, in the situation shown in FIG. 2B, theuser can currently see at least the surface features 204A and 204C onthe inside surface 214 and can therefore experience a cognitivelyplausible impression of being inside the AR object 200. One or moreeffects, filters and/or other modifications can be applied to the ARobject 200 as seen while the viewer 202 is on the inside. In someimplementations, a glass shader or a water shader can be applied. Forexample, the applied shader can provide the user with the visualimpression described above with regard to FIG. 1B.

As such, the above implementations can exemplify performance of a methodthat involves triggering rendering of an AR environment, such as the ARenvironment shown in FIGS. 1A-B. The AR environment can have the lightsource 208A and/or 208B and the viewer 202. The viewer 202 can be placedat various locations in the AR environment and can be defined forgenerating views of the AR environment from those locations. The ARobject 100 and/or 200 can be generated in the AR environment and canhave the outside surface 104 and/or 204 that is visualized using one ormore textures applied to one or more meshes. The mesh(es) can havedirections defined, such as by way of the normal vectors 206A-C whichcan be oriented away from the AR object 100/200. A determination can beperformed that the viewer 202 is inside the AR object 100/200 as aresult of movement in the AR environment. In response to such adetermination, a transparency of the outside surface 104/204 can beincreased. In response to such a determination, the direction of themesh(es) can be reversed, such as by generating the inverted normalvectors 206A′ and/or 206C′. In response to such a determination, theinside surface 214 of the AR object 100/200 can be rendered using atleast the part of the texture whose mesh has the reversed direction. Forexample, the surface features 204A and/or 204C can be rendered. Theinside surface 214 can be illuminated by the light rays 210A′ due to theincreased transparency. One advantage of such an approach can be that nochanges need to be made in the model of the AR object 100/200 itself,but rather the inside surface can be defined when needed (e.g., when itis determined that clipping occurs).

The viewer 202 may remain inside the AR object 200 indefinitely, or itmay again transition to the outside, such as by again clipping theoutside surface as a result of relative movement between the viewer 202and the AR object 200. Accordingly, the system that generates the ARenvironment can make a determination that the viewer 202 is now outsidethe AR object 200. In response to such a determination the transparencyof the outside surface 204 can be restored. For example, the AR object200 can be made non-transparent again. In response to such adetermination, the direction of the mesh(es) can be reverted to insteadbe directed away from the AR object 200. For example, the normal vectors206A-C can be restored.

FIG. 3 shows an example of reversing a direction of a mesh for an ARobject. Here, an object surface 300 is defined for the AR object, inanalogy with the outside surface 104 for the AR object 100 and/or withthe outside surface 204 for the AR object 200. As such, the entire ARobject is not currently show in the present figure, only the partcorresponding to the object surface 300. The AR object can be, or bepart of, any other AR object described herein, such as in FIG. 1A-B,2A-B, 4A-C or 5A-C. Here, the object surface 300 has an essentiallysquare configuration but can have other shapes in other implementations.

A mesh 302 is here defined for the object surface 300. For example, themesh 302 defines the shape or other properties of the object surface 300except for its visual appearance. Moreover, a material 304 is hereapplied to the mesh 302. The material 304 can have one or moreproperties, including a texture for the object surface 300. A directioncan be defined for the mesh 302, such as by way of one or more normalvectors 306. For example, the material 304 defines the appearance of theobject surface 300 when viewed from the outside of the AR object, suchas the appearance of the AR object 100 in FIG. 1A.

Similar to examples described above, a determination can be made that aviewer in the AR environment has clipped the object surface 300 of theAR object. For example, this determination can indicate that the vieweris currently positioned inside the AR object. In response to such adetermination, one or more operations can be performed. For example, thedirection of the mesh 302 can be reversed, such as by defining aninverted normal vector 306′. The definition of the inverted normalvector 306′ can effectively define a material 304′ on an inside surfaceof the AR object. For example, the material 304′ can allow the viewerinside the AR object to see, by way of the texture property, one or morefeatures that were previously defined as being on the outside surface ofthe AR object.

As another example, a shader can be applied to the material 304/304′ inresponse to determining that the viewer in the AR environment hasclipped the object surface 300 of the AR object. In someimplementations, a property of the material 304/304′ can be altered. Forexample, the shader can involve applying or modifying one or more of: aglass shader, a water shader, a translucency, a reflectiveness, arefraction, and/or a diffraction.

FIGS. 4A-C show an example of repositioning an AR object 400 based on aviewer position. The AR object 400 can be, or be part of, any other ARobject described herein, such as in FIG. 1A-B, 2A-B, 3 or 5A-C. The ARobject 400 is here defined in an AR environment 402. As with other ARenvironments, the AR environment 402 can include imagery of a physicalreality (e.g., a camera view of the user's surroundings) and imagery ofvirtual reality (e.g., the AR object 400). The presentation of the ARenvironment can then provide the user a view that simultaneously showsat least some of the imagery of the physical reality and at least someof the imagery of virtual reality. A viewer 404, such as any of theviewers described elsewhere herein, is defined in the AR environment402, and the viewer 404 can undergo movement, for example as indicatedby an arrow 406. As such, the relative movement between the viewer 404and the AR object 400 in the present example is based on the viewer 404moving and the AR object 400 initially being stationary. Another exampleof relative movement will be described below with reference to FIGS.12A-C.

In some implementations, a plausible interaction between the user(embodied by the viewer 404 defined in the AR environment 402) and theAR object 400 can be that the AR object 400 should in some sense behavelike an object in physical reality when interacted with by the user, butthat the interaction should not result in making any persistent changesin the appearance or location of the AR object 400.

The user can push the AR object 400 in one or more directions byabutting the viewer 404 against the outside surface of the AR object400. Here, the AR environment 402 has defined therein an axis 408 whichmay or may not be visible to the user. In some implementations, theposition of the AR object 400 can be defined using the axis 408. Theaxis 408 can quantify the movement of the AR object 400 with regard toan initial position P marked on the axis 408.

Assume now that the user pushes the AR object 400 using the viewer 404.For example, the user physically moves a handheld device which iscapturing the physical environment that is part of the AR environment402, or activates an input control on a device that controls the ARenvironment 402. FIG. 4B shows that the viewer 404 has pushed the ARobject 400 to a new position P′ on the axis 408. A dashed outline 400′indicates where in the AR environment 402 the AR object 400 wasinitially located and thus illustrates that the AR object 400 has beenmoved away from this position. During this interaction, the user maycontinue to see the outer surface of the AR object 400 through theviewer 404. This behavior can be consistent with the notion of pushingthe AR object 400 in front of oneself while moving. For example, alinear translation of the AR object 400 from the position P to theposition P′ can be performed as part of the interaction.

One or more thresholds can be established in the AR environment 402. Athreshold can define the maximum distance that the user can push theparticular AR object 400 in a given direction. In some implementations,the threshold can be defined based on a length of the AR object 400 in adirection of the continued movement of the viewer 404. For example, thethreshold can be reached when the AR object 400 has traveled a certainproportion of its own size, such as 50% of its length.

Assume now that the position P′ here corresponds to the threshold forthe particular AR object 400 in the current direction. In response tothe threshold being reached, the AR object 400 can be relocated to theinitial position P. FIG. 4C shows the AR environment 402 after the ARobject 400 is relocated to the position P along the axis 408 in responseto the threshold being reached. For example, the AR object 400 can betranslated in the opposite direction along the axis 408. That is, the ARobject 400 is now back at the initial position that it had before theinteraction, and the interaction has not produced any permanent orpersistent changes in the AR environment 402. However, the viewer 404 isnow on the opposite side of the AR object 400 than it was before andduring the course of the interaction. That is, when the AR object 400 isrelocated from the threshold position (e.g., the position P′) to theinitial position P, this can appear to the user as if the viewer 404 istraveling through (perhaps almost instantaneously) the AR object 400 andis emerging on the other side of the AR object 400. This can be aplausible behavior to the user and can give a cognitively connectedexperience of having been transported through the AR object 400.

As such, the above implementation shown in FIGS. 4A-C can exemplifyperformance of a method that involves rendering the AR environment 402.The AR environment 402 has the viewer 404 configured for movement tolocations in the AR environment 402 and for generating views of the ARenvironment 402 from the locations. The AR object 400 can also berendered in the AR environment 402. A first determination that theviewer 404 contacts the AR object 400 as a result of the movement can beperformed. In response to such a first determination, the AR object 400can be repositioned in the AR environment 402 to track continuedmovement of the viewer 404. For example, the AR object 400 can track themovement of the viewer 404 and thereby be repositioned from the positionP to the position P′. A second determination that the repositioning ofthe AR object 400—based on the continued movement of the viewer404—reaches a threshold such as the position P′ can be performed. Inresponse to such a second determination, the AR object 400 can berelocated to the position P without affecting the continued movement ofthe viewer 404. For example, as part of the relocation the viewer 404passes through the AR object 400 and emerges on the opposite sidethereof.

One or more effects can be applied to the AR object 400 as part of theinteraction. In some implementations, the effect is applied during thetime that the AR object 400 is displaced from its initial position. Anytype of effect can be applied, including but not limited to thosementioned elsewhere herein, such as a partial transparency. For example,this can allow the user to partially see through the AR object 400 whilepushing the AR object 400 in front of the user.

FIGS. 12A-C show another example of repositioning an AR object 1200based on a viewer position. The AR object 1200 can be, or be part of,any other AR object described herein, such as in FIG. 1A-B, 2A-B, 3,4A-C or 5A-C. The AR object 1200 is here defined in an AR environment1202. As with other AR environments, the AR environment 1202 can includeimagery of a physical reality (e.g., a camera view of the user'ssurroundings) and imagery of virtual reality (e.g., the AR object 1200).The presentation of the AR environment can then provide the user a viewthat simultaneously shows at least some of the imagery of the physicalreality and at least some of the imagery of virtual reality. A viewer1204, such as any of the viewers described elsewhere herein, is definedin the AR environment 1202. The AR object 1200 can undergo movementtoward the viewer 1204, for example as indicated by an arrow 1206. Assuch, the relative movement between the viewer 1204 and the AR object1200 in the present example is based on the AR object 1200 initiallymoving and the viewer 1204 being stationary.

In some implementations, a plausible interaction between the user(embodied by the viewer 1204 defined in the AR environment 1202) and theAR object 1200 can be that the AR object 1200 should in some sensebehave like an object in physical realty when interacted with by theuser, but that the interaction should not result in making anypersistent changes in the appearance or location of the AR object 1200.

The AR object 1200 can push against the viewer 1204 in one or moredirections by the outside surface of the AR object 1200 abutting againstthe viewer 1204. Here, the AR environment 1202 has defined therein anaxis 1208 which may or may not be visible to the user. In someimplementations, the position of the viewer 1204 and the AR object 1200can be defined using the axis 1208. The axis 1208 can quantify themovement of the AR object 1200 with regard to an initial position P ofthe viewer 1204 marked on the axis 1208.

Assume now that the AR object 1200 pushes against the viewer 1204. Forexample, this can occur because the AR object 1200 is defined to travelalong a path (e.g., a straight path) in the AR environment 1202. FIG.12B shows that the abutting of the AR object 1200 against the viewer1204 has temporarily halted the movement of the AR object 1200. A dashedoutline 1200′ indicates where in the AR environment 1202 the AR object1200 would have been located if it had not abutted the viewer 1204.During this interaction, the user may continue to see the outer surfaceof the AR object 1200 through the viewer 1204. This behavior can beconsistent with the notion of the AR object 1200 remaining in front ofthe viewer 1204 for a brief period of time.

One or more thresholds can be established in the AR environment 1202. Athreshold can define the maximum distance that the movement of the ARobject 1200 can be temporarily halted in a given direction as a resultof abutting the viewer 1204. Here, when the dashed outline 1200′ reachesa position P′ on the axis 1208, the temporary halting of the movement ofthe AR object 1200 can cease. In some implementations, the threshold canbe defined based on a length of the AR object 1200 in a direction towardthe viewer 1204. For example, the threshold can be reached when thedashed outline 1200′ has traveled a certain proportion of the size ofthe AR object 1200, such as 50% of its length.

Assume now that the position P′ here corresponds to the threshold forthe dashed outline 1200′ of the particular AR object 1200 in the currentdirection. In response to the threshold P′ being reached by the dashedoutline 1200′, the AR object 1200 can be relocated to a position P″ onthe other side of the viewer 1204, as indicated in FIG. 12C. Here, theAR environment 1202 is shown after the AR object 1200 is relocated tothe position P″ along the axis 1208 in response to the threshold P′being reached by the dashed outline 1200′. For example, the AR object1200 can be translated along the axis 1208. That is, the AR object 1200is now on the opposite side of the viewer 1204 compared to before theinteraction, and the interaction has not produced any permanent orpersistent changes in the AR environment 1202. That is, when the ARobject 1200 is relocated from the threshold position to the position P″,this can appear to the user as if the AR object 1200 is traveling past(perhaps almost instantaneously) the viewer 1204 and is emerging on theother side of the viewer 1204 (where it could be visible if the userturns the viewer 1204 in that direction). This can be a plausiblebehavior to the user and can give a cognitively connected experience ofthe AR object 1200 having been transported past the viewer 1204.

FIGS. 5A-C show an example of repositioning an AR object 500 based on aviewer position. The AR object 500 can be, or be part of, any other ARobject described herein, such as in FIG. 1A-B, 2A-B, 3 or 4A-C. The ARobject 500 is here defined in an AR environment 502. As with other ARenvironments, the AR environment 502 can include imagery of a physicalreality (e.g., a camera view of the user's surroundings) and imagery ofvirtual reality (e.g., the AR object 500). A viewer 504, such as any ofthe viewers described elsewhere herein, is defined in the AR environment502, and the viewer 504 can undergo movement, for example as indicatedby an arrow 506. As such, the relative movement between the viewer 504and the AR object 500 in the present example is based on the viewer 504moving and the AR object 500 initially being stationary. Another exampleof relative movement will be described below with reference to FIGS.13A-C.

In some implementations, a plausible interaction between the user(embodied by the viewer 504 defined in the AR environment 502) and theAR object 500 can be that the AR object 500 should in some sense behavelike an object in physical reality when interacted with by the user, butthat the interaction should not result in making any persistent changesin the appearance or location of the AR object 500.

The user can push the AR object 500 in one or more directions byabutting the viewer 504 against the outside surface of the AR object500. Here, the AR environment 502 has defined therein an axis 508 inanalogy with the axis 408 described above. Assume now that the userpushes the AR object 500 using the viewer 504. For example, the userphysically moves a handheld device which is capturing the physicalenvironment that is part of the AR environment 502, or activates aninput control on a device that controls the AR environment 502.

A fulcrum 510 is defined in the AR environment 502 in response to theinteraction between the viewer 504 and the AR object 500. A pivot arm512 is also defined in the AR environment 502 in response to theinteraction between the viewer 504 and the AR object 500. The fulcrum510 and/or the pivot arm 512 may not be visible to the user. Here, thepivot arm 512 connects the AR object 500 and the fulcrum 510 to eachother so that the AR object 500 is at least partially rotatable aboutthe fulcrum 510. In some implementations, the fulcrum 510 and the pivotarm 512 are not defined as part of the model that defines the AR object500 but rather are separate from the model of the AR object 500 and fromthe model of any other AR object that can be presented in the ARenvironment 502.

FIG. 5B shows that the viewer 504 has here pushed the AR object 500 asindicated by an arc 514 to a new position P′ on the axis 508. A dashedoutline 500′ indicates where in the AR environment 502 the AR object 500was initially located and thus illustrates that the AR object 500 hasbeen moved away from this position. During this interaction, the usermay continue to see the outer surface of the AR object 500 through theviewer 504. This behavior can be consistent with the notion of the ARobject 500 being suspended from the fulcrum 510, with the user pushingthe AR object 500 away (in this example, somewhat upward) by themovement in the AR environment 502. For example, a rotation of the ARobject 500 from the position P to the position P′ about the fulcrum 510can be performed as part of the interaction.

One or more thresholds can be established in the AR environment 502 inanalogy with the description above. For example, the threshold can bereached when the AR object 500 has traveled a certain proportion of itsown size, such as 50% of its length.

Assume now that the position P′ here corresponds to the threshold forthe particular AR object 500 in the current direction. In response tothe threshold being reached, the AR object 500 can be relocated to theinitial position P. FIG. 5C shows the AR environment 502 after the ARobject 500 is relocated to the position P along the axis 508 in responseto the threshold being reached. For example, the AR object 500 can berotated in the opposite direction about the fulcrum 510. That is, the ARobject 500 is now back at the initial position that it had before theinteraction, and the interaction has not produced any permanent orpersistent changes in the AR environment 502. However, the viewer 504 isnow on the opposite side of the AR object 500 than it was before andduring the course of the interaction. That is, when the AR object 500 isrelocated from the threshold position (e.g., the position P′) to theinitial position P, this can appear to the user as if the viewer 504 istraveling through (perhaps almost instantaneously) the AR object 500 andis emerging on the other side of the AR object 500. This can be aplausible behavior to the user and can give a cognitively connectedexperience of having been transported through the AR object 500.

As such, the above implementation shown in FIGS. 5A-C can exemplifyperformance of a method that involves rendering the AR environment 502.The AR environment 502 has the viewer 504 configured for movement tolocations in the AR environment 502 and for generating views of the ARenvironment 502 from the locations. The AR object 500 can also berendered in the AR environment 502. A first determination that theviewer 504 contacts the AR object 500 as a result of the movement can beperformed. In response to such a first determination, the AR object 500can be repositioned in the AR environment 502 to track continuedmovement of the viewer 504. For example, the AR object 500 can track themovement of the viewer 504 and thereby be repositioned from the positionP to the position P′. A second determination that the repositioning ofthe AR object 500—based on the continued movement of the viewer504—reaches a threshold such as the position P′ can be performed. Inresponse to such a second determination, the AR object 500 can berelocated to the position P without affecting the continued movement ofthe viewer 504. For example, as part of the relocation the viewer 504passes through the AR object 500 and emerges on the opposite sidethereof.

One or more effects can be applied to the AR object 500 as part of theinteraction. In some implementations, the effect is applied during thetime that the AR object 500 is displaced from its initial position. Anytype of effect can be applied, including but not limited to thosementioned elsewhere herein, such as a partial transparency. For example,this can allow the user to partially see through the AR object 500 whilepushing the suspended AR object 500 in front of the user.

In the above example, the fulcrum 510 was defined essentially above theAR object 500 so that the AR object 500 in a sense can be considered tobe suspended from the fulcrum 510 during the interaction with the viewer504. For example, the interaction with the AR object 500 can then beanalogous with the behavior of an object such as a punching bag that issuspended from the ceiling. In some implementations, the fulcrum 510 (oranother fulcrum) can be defined essentially below the AR object 500 sothat the AR object 500 in a sense can be considered to be tethered tothe ground by the pivot arm 512 during the interaction with the viewer504. For example, the interaction with the AR object 500 can then beanalogous with the behavior of a helium balloon tied to a string.

FIGS. 13A-C show another example of repositioning an AR object 1300based on a viewer position. The AR object 1300 can be, or be part of,any other AR object described herein, such as in FIG. 1A-B, 2A-B, 3,4A-C or 5A-C. The AR object 1300 is here defined in an AR environment1302. As with other AR environments, the AR environment 1302 can includeimagery of a physical reality (e.g., a camera view of the user'ssurroundings) and imagery of virtual reality (e.g., the AR object 1300).A viewer 1304, such as any of the viewers described elsewhere herein, isdefined in the AR environment 1302. The AR object 1300 can undergomovement, for example as indicated by an arrow 1306. As such, therelative movement between the viewer 1304 and the AR object 1300 in thepresent example is based on the AR object 1300 moving and the viewer1304 being stationary.

In some implementations, a plausible interaction between the user(embodied by the viewer 1304 defined in the AR environment 1302) and theAR object 1300 can be that the AR object 1300 should in some sensebehave like an object in physical reality when interacted with by theuser, but that the interaction should not result in making anypersistent changes in the appearance or location of the AR object 1300.

The AR object 1300 can push against the viewer 1304 in one or moredirections by the outside surface of the AR object 1300 abutting againstthe viewer 1304. Here, the AR environment 1302 has defined therein anaxis 1308 in analogy with the axis 508 described above.

A fulcrum 1310 is defined in the AR environment 1302 in response to theinteraction between the AR object 1300 and the viewer 1304. A pivot arm1312 is also defined in the AR environment 1302 in response to theinteraction between the AR object 1300 and the viewer 1304. The fulcrum1310 and/or the pivot arm 1312 may not be visible to the user. Here, thepivot arm 1312 connects the AR object 1300 and the fulcrum 1310 to eachother so that the AR object 1300 is at least partially rotatable aboutthe fulcrum 1310. In some implementations, the fulcrum 1310 and thepivot arm 1312 are not defined as part of the model that defines the ARobject 1300 but rather are separate from the model of the AR object 1300and from the model of any other AR object that can be presented in theAR environment 1302.

Assume now that the AR object 1300 pushes against the viewer 1304 whichis positioned at the position P on the axis 1308. For example, this canoccur because the AR object 1300 is defined to travel along a path(e.g., a straight path) in the AR environment 1302. FIG. 13B shows thatthe abutting of the AR object 1300 against the viewer 1304 has herepushed the AR object 1300 as indicated by an arc 1314. During thisinteraction, the user may continue to see the outer surface of the ARobject 1300 through the viewer 1304. This behavior can be consistentwith the notion of the AR object 1300 being suspended from the fulcrum1310 during its travel, with the AR object 1300 being pushed away (inthis example, somewhat upward) by the fact that it abuts against theviewer 1304 on the course of its movement in the AR environment 1302.For example, a rotation of the AR object 1300 about the fulcrum 1310 canbe performed as part of the interaction.

One or more thresholds can be established in the AR environment 1302 inanalogy with the description above. A threshold can define the maximumdistance that the fulcrum 1310 can travel while the AR object 1300 isabutting against the viewer 1304. Here, when the fulcrum 1310 reaches aposition P′ on the axis 1308, the abutting of the AR object 1300 againstthe viewer 1304 can cease.

Assume now that the position P′ here corresponds to the threshold forthe fulcrum 1310 of the particular AR object 1300 in the currentdirection. In response to the threshold P′ being reached by the fulcrum1310, the AR object 1300 can be relocated to a position P″ on the otherside of the viewer 1304, as indicated in FIG. 13C. Here, the ARenvironment 1302 is shown after the AR object 1300 is relocated to theposition P″ along the axis 1308 in response to the threshold beingreached by the fulcrum 1310. For example, the AR object 1300 can berotated in the opposite direction about the fulcrum 1310. That is, theAR object 1300 is now on the opposite side of the viewer 1304 comparedto before the interaction, and the interaction has not produced anypermanent or persistent changes in the AR environment 1302. That is,when the AR object 1300 is relocated from the threshold position to theposition P″, this can appear to the user as if the AR object 1300 istraveling past (perhaps almost instantaneously) the viewer 1304 and isemerging on the other side of the viewer 1304 (where it could be visibleif the user turns the viewer 1304 in that direction). This can be aplausible behavior to the user and can give a cognitively connectedexperience of having been transported past the AR object 1300.

One or more aspects of the above examples can be defined as ageneralized behavior in the system that generates an AR environment.This can reduce or eliminate the need to modify the models correspondingto AR objects, or to define object interactions separately for each ARobject. Rather, the AR environment can have predefined therein one ormore modes of interaction that can then generically be applied to anyand all AR objects that are rendered. For example, referring againbriefly to FIGS. 1A-B, the AR object 100 can be created with only thetexture for the outside surface 104, and not any inside texture. When itis determined that the viewer clips the outside surface 104, the systemcan dynamically and in real time perform the reversal of direction ofthe mesh of the texture for the outside surface 104 so as to obtain thetexture for the inside surface. As another example, referring againbriefly to FIG. 4A-C (or 5A-C), the behavior that the AR object 400 (or500) is temporarily pushed away by the viewer 404 (or 504) up until athreshold, and thereafter returns to its original position, can be ageneralized behavior that is defined in the AR environment as beingpotentially applicable to all AR objects or other properties therein.For example, the fulcrum 510 and the pivot arm 512 may not constantly bepresent in the

FIGS. 6-10 show examples of methods. Any of the methods can be performedin one or more computer systems, such as by at least one processorexecuting instructions stored in a non-transitory medium. For example,the method(s) can be performed in the device(s) shown in FIG. 11. Moreor fewer operations than shown can be performed with any or all off themethods. Also, two or more operations can be performed in a differentorder.

FIG. 6 shows an example of a method 600 that involves interactionbetween an AR viewer and an object in an AR environment. The AR objectand/or the AR environment can be any of those described elsewhereherein, such as in FIG. 1A-B, 2A-B, 3, 4A-C or 5A-C. At 602, an ARenvironment can be generated. At 604, rendering of the AR object can betriggered. At 606, the AR viewer can be activated. For example, thisallows the user to see the AR environment. At 608, the AR viewer can benavigated within the AR environment. For example, the user controlswhere the AR viewer should be moved. At 610, the system can detect thatthe AR viewer clips an outside surface of the AR object. At 612, one ormore effects, including but not limited to transparency, can be appliedto the AR object. At 614, the direction of one or more meshes can bereversed, such as by inverting a normal vector of the mesh. At 616, afilter or other effect can be applied to the AR object, including, butnot limited to, a glass effect or a water effect. At 618, rendering ofat least part of the inside of the AR object can be triggered. Forexample, the increased transparency of the outside surface may providevirtual light that illuminates the inside surface of the AR object andthereby makes it observable to the user. At 620, the user can (continueto) navigate the viewer. For example, once inside the AR object the usercan continue to move in the same direction, or the user can turn theviewer and move in a different direction. At 622, the system can detectthat the viewer clips the outside surface of the AR object and isthereafter positioned outside the AR object. In response to thedetection of the clipping, the transparency can be removed (at 624), thenormal vector(s) can be reverted (at 626), and/or the filter(s) can beremoved (at 628). At 630, rendering of the AR object or another aspectof the AR environment can be triggered.

FIG. 7 shows examples of effects, filters and other features that can beapplied to an AR object in a method 700. One or more operations of themethod 700 can be applied during performance of another method describedherein, including, but not limited to, as part of operation 612 and/or618 in FIG. 6. Some examples of components from other implementationsdescribed herein will be mentioned solely for purposes of explanation.For example, one or more of the operations in the method 700 can beapplied to the mesh 302 (FIG. 3) of an AR object.

At 710, a glass shader can be applied. For example, the glass effect canmake the AR object 100 (FIG. 1B) appear as if it were made of glass.

At 720, a water effect can be applied. For example, the water effect canmake the AR object 100 (FIG. 1B) appear as if it were made of water.

At 730, a translucency can be applied. For example, the translucency canmake the AR object 100 (FIG. 1B) translucent.

At 740, a reflectiveness can be altered. For example, the change canmake the AR object 100 (FIG. 1B) more reflective or less reflective.

At 750, a refraction can be altered. For example, the change can makethe AR object 100 (FIG. 1B) more refractive or less refractive.

At 760, a Fresnel diffraction can be applied. In some implementations, aFresnel shader is applied. For example, the Fresnel diffraction cangenerate one or more diffraction patterns on the AR object 100 (FIG.1B).

FIG. 8 shows a method 800 that is an example of temporarily displacingan AR object as part of an interaction with a viewer in an ARenvironment. The AR object and/or the AR environment can be any of thosedescribed elsewhere herein, such as in FIG. 1A-B, 2A-B, 3, 4AC or 5A-C.At 802, an AR environment can be generated.

At 804, rendering of an AR object can be triggered. At 806, an AR viewercan be activated. At 808, the AR viewer can be navigated by a user. At810, contact between the AR viewer and the AR object can be detected. Inresponse to the detection, the AR object can be relocated in the ARenvironment at 812. At 814, the user can (continue to) navigate the ARviewer in the AR environment. At 816, the AR object can be repositionedbased on the position of the AR viewer. At 818, rendering of therepositioned object can be triggered. For example, the AR object can becontinuously rendered during the repositioning performed at 816. At 820,it can be determined that the repositioning of the AR object has atleast reached a threshold. At 822, and in response to the determinationat 820, the AR object can be repositioned to its initial position.

FIG. 9 shows an example of a method 900 of linearly translating an ARobject. The AR object can be any of those described elsewhere herein,such as in FIG. 1A-B, 2A-B, 3, 4AC or 5A-C. One or more operations ofthe method 900 can be applied during performance of another methoddescribed herein, including, but not limited to, as part of operations810-20 in FIG. 8. At 910, it can be determined that an AR viewercontacts the AR object in an AR environment. At 920, the (continued)movement of the AR viewer can be tracked. At 930, a translation of theAR object can be performed. The translation can be linear and cancorrespond to the movement of the AR viewer. At 940, an effect can beapplied to the AR object, including, but not limited to, a partialtransparency. At 950, rendering of the altered (e.g., partiallytransparent) AR object can be triggered so as to be visible to the user.At 960, it can be determined that a threshold in the AR environment hasbeen reached.

FIG. 10 shows an example of a method 1000 of linearly translating an ARobject. The AR object can be any of those described elsewhere herein,such as in FIG. 1A-B, 2A-B, 3, 4AC or 5A-C. One or more operations ofthe method 1000 can be applied during performance of another methoddescribed herein, including, but not limited to, as part of operations810-20 in FIG. 8. At 1010, it can be determined that an AR viewercontacts the AR object in an AR environment. At 1020, a fulcrum andpivot arm for the AR object can be defined in response to the detection.At 1030, the (continued) movement of the AR viewer can be tracked. At1040, a rotation of the AR object can be performed. The rotation can beof the AR object and the pivot arm about the fulcrum, and can correspondto the movement of the AR viewer. At 1050, an effect can be applied tothe AR object, including, but not limited to, a partial transparency. At1060, rendering of the altered (e.g., partially transparent) AR objectcan be triggered so as to be visible to the user. At 1070, it can bedetermined that a threshold in the AR environment has been reached.

FIG. 11 shows an example of a computer device and a mobile computerdevice that can be used to implement the techniques described here. FIG.11 shows an example of a generic computer device 1100 and a genericmobile computer device 1150, which may be used with the techniquesdescribed here. Computing device 1100 is intended to represent variousforms of digital computers, such as laptops, desktops, tablets,workstations, personal digital assistants, televisions, servers, bladeservers, mainframes, and other appropriate computing devices. Computingdevice 1150 is intended to represent various forms of mobile devices,such as personal digital assistants, cellular telephones, smart phones,and other similar computing devices. The components shown here, theirconnections and relationships, and their functions, are meant to beexemplary only, and are not meant to limit implementations of theinventions described and/or claimed in this document.

Computing device 1100 includes a processor 1102, memory 1104, a storagedevice 1106, a high-speed interface 1108 connecting to memory 1104 andhigh-speed expansion ports 1110, and a low speed interface 1112connecting to low speed bus 1114 and storage device 1106. The processor1102 can be a semiconductor-based processor. The memory 1104 can be asemiconductor-based memory. Each of the components 1102, 1104, 1106,1108, 1110, and 1112, are interconnected using various busses, and maybe mounted on a common motherboard or in other manners as appropriate.The processor 1102 can process instructions for execution within thecomputing device 1100, including instructions stored in the memory 1104or on the storage device 1106 to display graphical information for a GUIon an external input/output device, such as display 1116 coupled to highspeed interface 1108. In other implementations, multiple processorsand/or multiple buses may be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices 1100 maybe connected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory 1104 stores information within the computing device 1100. Inone implementation, the memory 1104 is a volatile memory unit or units.In another implementation, the memory 1104 is a non-volatile memory unitor units. The memory 1104 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 1106 is capable of providing mass storage for thecomputing device 1100. In one implementation, the storage device 1106may be or contain a computer-readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product can be tangibly embodied inan information carrier. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The information carrier is a computer- ormachine-readable medium, such as the memory 1104, the storage device1106, or memory on processor 1102.

The high speed controller 1108 manages bandwidth-intensive operationsfor the computing device 1100, while the low speed controller 1112manages lower bandwidth-intensive operations. Such allocation offunctions is exemplary only. In one implementation, the high-speedcontroller 1108 is coupled to memory 1104, display 1116 (e.g., through agraphics processor or accelerator), and to high-speed expansion ports1110, which may accept various expansion cards (not shown). In theimplementation, low-speed controller 1112 is coupled to storage device1106 and low-speed expansion port 1114. The low-speed expansion port,which may include various communication ports (e.g., USB, Bluetooth,Ethernet, wireless Ethernet) may be coupled to one or more input/outputdevices, such as a keyboard, a pointing device, a scanner, or anetworking device such as a switch or router, e.g., through a networkadapter.

The computing device 1100 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 1120, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system 1124. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 1122. Alternatively, components from computing device 1100 maybe combined with other components in a mobile device (not shown), suchas device 1150. Each of such devices may contain one or more ofcomputing device 1100, 1150, and an entire system may be made up ofmultiple computing devices 1100, 1150 communicating with each other.

Computing device 1150 includes a processor 1152, memory 1164, aninput/output device such as a display 1154, a communication interface1166, and a transceiver 1168, among other components. The device 1150may also be provided with a storage device, such as a microdrive orother device, to provide additional storage. Each of the components1150, 1152, 1164, 1154, 1166, and 1168, are interconnected using variousbuses, and several of the components may be mounted on a commonmotherboard or in other manners as appropriate.

The processor 1152 can execute instructions within the computing device1150, including instructions stored in the memory 1164. The processormay be implemented as a chipset of chips that include separate andmultiple analog and digital processors. The processor may provide, forexample, for coordination of the other components of the device 1150,such as control of user interfaces, applications run by device 1150, andwireless communication by device 1150.

Processor 1152 may communicate with a user through control interface1158 and display interface 1156 coupled to a display 1154. The display1154 may be, for example, a TFT LCD (Thin-Film-Transistor Liquid CrystalDisplay) or an OLED (Organic Light Emitting Diode) display, or otherappropriate display technology. The display interface 1156 may compriseappropriate circuitry for driving the display 1154 to present graphicaland other information to a user. The control interface 1158 may receivecommands from a user and convert them for submission to the processor1152. In addition, an external interface 1162 may be provide incommunication with processor 1152, so as to enable near areacommunication of device 1150 with other devices. External interface 1162may provide, for example, for wired communication in someimplementations, or for wireless communication in other implementations,and multiple interfaces may also be used.

The memory 1164 stores information within the computing device 1150. Thememory 1164 can be implemented as one or more of a computer-readablemedium or media, a volatile memory unit or units, or a non-volatilememory unit or units. Expansion memory 1174 may also be provided andconnected to device 1150 through expansion interface 1172, which mayinclude, for example, a SIMM (Single In Line Memory Module) cardinterface. Such expansion memory 1174 may provide extra storage spacefor device 1150, or may also store applications or other information fordevice 1150. Specifically, expansion memory 1174 may includeinstructions to carry out or supplement the processes described above,and may include secure information also. Thus, for example, expansionmemory 1174 may be provide as a security module for device 1150, and maybe programmed with instructions that permit secure use of device 1150.In addition, secure applications may be provided via the SIMM cards,along with additional information, such as placing identifyinginformation on the SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 1164, expansionmemory 1174, or memory on processor 1152, that may be received, forexample, over transceiver 1168 or external interface 1162.

Device 1150 may communicate wirelessly through communication interface1166, which may include digital signal processing circuitry wherenecessary. Communication interface 1166 may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication may occur, for example, through radio-frequencytransceiver 1168. In addition, short-range communication may occur, suchas using a Bluetooth, WiFi, or other such transceiver (not shown). Inaddition, GPS (Global Positioning System) receiver module 1170 mayprovide additional navigation- and location-related wireless data todevice 1150, which may be used as appropriate by applications running ondevice 1150.

Device 1150 may also communicate audibly using audio codec 1160, whichmay receive spoken information from a user and convert it to usabledigital information. Audio codec 1160 may likewise generate audiblesound for a user, such as through a speaker, e.g., in a handset ofdevice 1150. Such sound may include sound from voice telephone calls,may include recorded sound (e.g., voice messages, music files, etc.) andmay also include sound generated by applications operating on device1150.

The computing device 1150 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone 1180. It may also be implemented as part of a smartphone 1182, personal digital assistant, or other similar mobile device.

A user can interact with a computing device using a tracked controller1184. In some implementations, the controller 1184 can track themovement of a user's body, such as of the hand, foot, head and/or torso,and generate input corresponding to the tracked motion. The input cancorrespond to the movement in one or more dimensions of motion, such asin three dimensions. For example, the tracked controller can be aphysical controller for a VR application, the physical controllerassociated with one or more virtual controllers in the VR application.As another example, the controller 1184 can include a data glove.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium”“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

In some implementations, the computing devices depicted in FIG. 11 caninclude sensors that interface with a virtual reality (VR headset 1185).For example, one or more sensors included on a computing device 1150 orother computing device depicted in FIG. 11, can provide input to VRheadset 1185 or in general, provide input to a VR space. The sensors caninclude, but are not limited to, a touchscreen, accelerometers,gyroscopes, pressure sensors, biometric sensors, temperature sensors,humidity sensors, and ambient light sensors. The computing device 1150can use the sensors to determine an absolute position and/or a detectedrotation of the computing device in the VR space that can then be usedas input to the VR space. For example, the computing device 1150 may beincorporated into the VR space as a virtual object, such as acontroller, a laser pointer, a keyboard, a weapon, etc. Positioning ofthe computing device/virtual object by the user when incorporated intothe VR space can allow the user to position the computing device to viewthe virtual object in certain manners in the VR space. For example, ifthe virtual object represents a laser pointer, the user can manipulatethe computing device as if it were an actual laser pointer. The user canmove the computing device left and right, up and down, in a circle,etc., and use the device in a similar fashion to using a laser pointer.

In some implementations, one or more input devices included on, orconnect to, the computing device 1150 can be used as input to the VRspace. The input devices can include, but are not limited to, atouchscreen, a keyboard, one or more buttons, a trackpad, a touchpad, apointing device, a mouse, a trackball, a joystick, a camera, amicrophone, earphones or buds with input functionality, a gamingcontroller, or other connectable input device. A user interacting withan input device included on the computing device 1150 when the computingdevice is incorporated into the VR space can cause a particular actionto occur in the VR space.

In some implementations, a touchscreen of the computing device 1150 canbe rendered as a touchpad in VR space. A user can interact with thetouchscreen of the computing device 1150. The interactions are rendered,in VR headset 1185 for example, as movements on the rendered touchpad inthe VR space. The rendered movements can control objects in the VRspace.

In some implementations, one or more output devices included on thecomputing device 1150 can provide output and/or feedback to a user ofthe VR headset 1185 in the VR space. The output and feedback can bevisual, tactical, or audio. The output and/or feedback can include, butis not limited to, vibrations, turning on and off or blinking and/orflashing of one or more lights or strobes, sounding an alarm, playing achime, playing a song, and playing of an audio file. The output devicescan include, but are not limited to, vibration motors, vibration coils,piezoelectric devices, electrostatic devices, light emitting diodes(LEDs), strobes, and speakers.

In some implementations, the computing device 1150 may appear as anotherobject in a computer-generated, 3D environment. Interactions by the userwith the computing device 1150 (e.g., rotating, shaking, touching atouchscreen, swiping a finger across a touch screen) can be interpretedas interactions with the object in the VR space. In the example of thelaser pointer in a VR space, the computing device 1150 appears as avirtual laser pointer in the computer-generated, 3D environment. As theuser manipulates the computing device 1150, the user in the VR spacesees movement of the laser pointer. The user receives feedback frominteractions with the computing device 1150 in the VR space on thecomputing device 1150 or on the VR headset 1185.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications may be made without departing fromthe spirit and scope of the invention.

In addition, the logic flows depicted in the figures do not require theparticular order shown, or sequential order, to achieve desirableresults. In addition, other steps may be provided, or steps may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherembodiments are within the scope of the following claims.

What is claimed is:
 1. A method comprising: triggering rendering of anaugmented reality (AR) environment having a virtual viewer configuredfor generating views of the AR environment; triggering rendering of avirtual object at a position in the AR environment; performing a firstdetermination that the virtual viewer and the virtual object contacteach other as a result of relative movement between the virtual viewerand the virtual object; in response to the first determination,repositioning the virtual object in the AR environment based on relativemovement between the virtual viewer and the virtual object, ortemporarily halting a movement of the virtual object; performing asecond determination that the repositioning of the virtual object basedon the relative movement, or the temporary halting of the movement ofthe virtual object, reaches a threshold; and in response to the seconddetermination, relocating the virtual object.
 2. The method of claim 1,wherein the threshold is defined based on a length of the virtual objectin a direction of the relative movement.
 3. The method of claim 1,wherein the repositioning comprises performing a linear translation ofthe virtual object according to a continued movement of the virtualviewer.
 4. The method of claim 1, further comprising positioning, inresponse to the first determination, a fulcrum in the AR environment,and defining a pivot arm from the virtual object to the fulcrum, whereinthe repositioning of the object comprises rotating the virtual objectand the pivot arm about the fulcrum.
 5. The method of claim 4, whereinthe fulcrum is defined above the virtual object in the AR environment.6. The method of claim 4, wherein the fulcrum is defined below thevirtual object in the AR environment.
 7. The method of claim 4, whereinthe positioning of the fulcrum and definition of the arm are performedaccording to a generalized behavior defined for all properties in the ARenvironment.
 8. The method of claim 4, further comprising moving thefulcrum along an original direction of the movement of the virtualobject.
 9. The method of claim 8, wherein the threshold comprises thatthe fulcrum reaches a position.
 10. The method of claim 8, whereinrelocating the virtual object results in the virtual viewer being on anopposite side of the virtual object than the virtual viewer was beforethe first determination.
 11. The method of claim 1, wherein relocatingthe virtual object results in the virtual viewer being on an oppositeside of the virtual object than the virtual viewer was before the firstdetermination and during the repositioning of the virtual object. 12.The method of claim 1, further comprising applying an effect to thevirtual object in response to the first determination, and removing theeffect in response to the second determination.
 13. The method of claim12, wherein applying the effect comprises altering a transparency of thevirtual object, and wherein removing the effect comprises restoring thetransparency of the virtual object.
 14. A system comprising: aprocessor; a display coupled to the processor; and a non-transitorystorage medium coupled to the processor and having stored thereoninstructions that when executed by the processor are configured to causethe processor to perform operations including: triggering rendering onthe display of an augmented reality (AR) environment having a virtualviewer configured for generating views of the AR environment; triggeringrendering on the display of a virtual object at a position in the ARenvironment; performing a first determination that the virtual viewerand the virtual object contact each other as a result of relativemovement between the virtual viewer and the virtual object; in responseto the first determination, repositioning the virtual object in the ARenvironment based on relative movement between the virtual viewer andthe virtual object, or temporarily halting a movement of the virtualobject; performing a second determination that the repositioning of thevirtual object based on the relative movement, or the temporary haltingof the movement of the virtual object, reaches a threshold; and inresponse to the second determination, relocating the virtual object. 15.The system of claim 14, wherein the relative movement between thevirtual viewer and the virtual object comprises movement of the virtualviewer toward the virtual object.
 16. The system of claim 15, whereinrelocating the virtual object results in the virtual viewer being on anopposite side of the virtual object than the virtual viewer was beforethe first determination.
 17. The system of claim 14, wherein therelative movement between the virtual viewer and the virtual objectcomprises movement of the virtual object toward the virtual viewer. 18.The system of claim 14, the operations further comprising positioning,in response to the first determination, a fulcrum in the AR environment,and defining a pivot arm from the virtual object to the fulcrum, whereinthe repositioning of the object comprises rotating the virtual objectand the pivot arm about the fulcrum.
 19. The system of claim 17, whereinrelocating the virtual object results in the virtual object being on anopposite side of the virtual viewer than the virtual object was beforethe first determination.
 20. A non-transitory storage medium havingstored thereon instructions that when executed are configured to cause aprocessor to perform operations, the operations comprising: triggeringrendering of an augmented reality (AR) environment having a virtualviewer configured for generating views of the AR environment; triggeringrendering of a virtual object at a position in the AR environment;performing a first determination that the virtual viewer and the virtualobject contact each other as a result of relative movement between thevirtual viewer and the virtual object; in response to the firstdetermination, repositioning the virtual object in the AR environmentbased on relative movement between the virtual viewer and the virtualobject, or temporarily halting a movement of the virtual object;performing a second determination that the repositioning of the virtualobject based on the relative movement, or the temporary halting of themovement of the virtual object, reaches a threshold; and in response tothe second determination, relocating the virtual object.